Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package

Caryophyllaeid Cestodes from Four Species of Carpiodes (Teleostei: Catostomidae)

The caryophyllaeid cestode fauna of four species of carpsuckers was investigated. Four hundred and thirty hosts from Iowa, Minnesota, Wisconsin, and Nebraska were examined (Aug. 1967-Dec. 1968) and 260 (60%) were parasitized. Four species of caryophyllaeids were found, of which Spartoides wardi and Biacetabulum carpiodi were most abundant. B. carpiodi exhibits a definite seasonal periodicity in spring and early summer, but none appears to exist for S. wardi. Single infections of Glaridacris confusa and Monobothrium sp. were also encountered

deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks

A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endangers true progress, as seeming improvements over a baseline might be statistical flukes, leading follow-up research astray while wasting human and computational resources. Here, we provide an easy-to-use package containing different significance tests and utility functions specifically tailored towards research needs and usability

On the Realization of Compositionality in Neural Networks

We present a detailed comparison of two types of sequence to sequence models trained to conduct a compositional task. The models are architecturally identical at inference time, but differ in the way that they are trained: our baseline model is trained with a task-success signal only, while the other model receives additional supervision on its attention mechanism (Attentive Guidance), which has shown to be an effective method for encouraging more compositional solutions (Hupkes et al.,2019). We first confirm that the models with attentive guidance indeed infer more compositional solutions than the baseline, by training them on the lookup table task presented by Li\v{s}ka et al. (2019). We then do an in-depth analysis of the structural differences between the two model types, focusing in particular on the organisation of the parameter space and the hidden layer activations and find noticeable differences in both these aspects. Guided networks focus more on the components of the input rather than the sequence as a whole and develop small functional groups of neurons with specific purposes that use their gates more selectively. Results from parameter heat maps, component swapping and graph analysis also indicate that guided networks exhibit a more modular structure with a small number of specialized, strongly connected neurons.Comment: To appear at BlackboxNLP 2019, AC

Experimental Standards for Deep Learning Research: A Natural Language Processing Perspective

The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and support scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs

Uncertainty in Natural Language Generation: From Theory to Applications

Recent advances of powerful Language Models have allowed Natural Language Generation (NLG) to emerge as an important technology that can not only perform traditional tasks like summarisation or translation, but also serve as a natural language interface to a variety of applications. As such, it is crucial that NLG systems are trustworthy and reliable, for example by indicating when they are likely to be wrong; and supporting multiple views, backgrounds and writing styles -- reflecting diverse human sub-populations. In this paper, we argue that a principled treatment of uncertainty can assist in creating systems and evaluation protocols better aligned with these goals. We first present the fundamental theory, frameworks and vocabulary required to represent uncertainty. We then characterise the main sources of uncertainty in NLG from a linguistic perspective, and propose a two-dimensional taxonomy that is more informative and faithful than the popular aleatoric/epistemic dichotomy. Finally, we move from theory to applications and highlight exciting research directions that exploit uncertainty to power decoding, controllable generation, self-assessment, selective answering, active learning and more

Harmonizing Lipidomics: NIST Interlaboratory Comparison Exercise for Lipidomics Using SRM 1950-metabolites in Frozen Human Plasma

As the lipidomics field continues to advance, self-evaluation within the community is critical. Here, we performed an interlaboratory comparison exercise for lipidomics using Standard Reference Material (SRM) 1950-Metabolites in Frozen Human Plasma, a commercially available reference material. The interlaboratory study comprised 31 diverse laboratories, with each laboratory using a different lipidomics workflow. A total of 1,527 unique lipids were measured across all laboratories and consensus location estimates and associated uncertainties were determined for 339 of these lipids measured at the sum composition level by five or more participating laboratories. These evaluated lipids detected in SRM 1950 serve as community-wide benchmarks for intra-and interlaboratory quality control and method validation. These analyses were performed using nonstandardized laboratory-independent workflows. The consensus locations were also compared with a previous examination of SRM 1950 by the LIPID MAPS consortium.jlr While the central theme of the interlaboratory study was to provide values to help harmonize lipids, lipid mediators, and precursor measurements across the community, it was also initiated to stimulate a discussion regarding areas in need of improvement

State-of-the-art generalisation research in NLP: a taxonomy and review

The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what `good generalisation' entails and how it should be evaluated is not well understood, nor are there any common standards to evaluate it. In this paper, we aim to lay the ground-work to improve both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP, we use that taxonomy to present a comprehensive map of published generalisation studies, and we make recommendations for which areas might deserve attention in the future. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they aim to solve, the type of data shift they consider, the source by which this data shift is obtained, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 previous papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis of the current state of generalisation research in NLP, and make recommendations for the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to up-date as new NLP generalisation studies are published. With this work, we aim to make steps towards making state-of-the-art generalisation testing the new status quo in NLP.Comment: 35 pages of content + 53 pages of reference

A qualitative investigation of breast cancer survivors’ experiences with breastfeeding

This is an exploratory, qualitative investigation of breast cancer survivors’ experiences with breastfeeding. Previous studies have focused on the physiology of lactation after surgery and treatment, but have not explored factors influencing breastfeeding decisions and behavior. We used purposeful sampling to identify 11 breast cancer survivors who had a child after their diagnosis and treatment. Participants were recruited from among those in the Women’s Healthy Eating and Living (WHEL) study and a Young Survival Coalition (YSC) affiliate. We conducted semi-structured, open-ended telephone interviews lasting 45–75 min. We used social cognitive theory (SCT) to structure questions regarding influences on breastfeeding behavior. We transcribed interviews and used cross-case, inductive analysis to identify themes. Ten of 11 participants initiated breastfeeding. The following main themes emerged: 1) Cautiously hopeful, 2) Exhausting to rely on one breast, 3) Motivated despite challenges, 4) Support and lack of support, and 5) Encouraging to others. Study participants were highly motivated to breastfeed but faced considerable challenges. Participants described problems that are not unique to women with breast cancer, but experienced these to a much greater degree because they relied mostly or entirely on one lactating breast. This study revealed a need for improved access to information and support and greater sensitivity to the obstacles faced by breast cancer survivors. Results of this qualitative analysis indicate that interventions to support the efforts of breast cancer survivors who are interested in breastfeeding are warranted. Additional research would aid in the development of such interventions

Skin Cancer:Epidemiology, Disease Burden, Pathophysiology, Diagnosis, and Therapeutic Approaches

Skin cancer, including both melanoma and non-melanoma, is the most common type of malignancy in the Caucasian population. Firstly, we review the evidence for the observed increase in the incidence of skin cancer over recent decades, and investigate whether this is a true increase or an artefact of greater screening and over-diagnosis. Prevention strategies are also discussed. Secondly, we discuss the complexities and challenges encountered when diagnosing and developing treatment strategies for skin cancer. Key case studies are presented that highlight the practic challenges of choosing the most appropriate treatment for patients with skin cancer. Thirdly, we consider the potential risks and benefits of increased sun exposure. However, this is discussed in terms of the possibility that the avoidance of sun exposure in order to reduce the risk of skin cancer may be less important than the reduction in all-cause mortality as a result of the potential benefits of increased exposure to the sun. Finally, we consider common questions on human papillomavirus infection